1 Convergence Theorem

1.1 LLN

Suppose we have ${\displaystyle X_1,\cdots ,X_n}$, where ${\displaystyle n}$ is very large. We are curious about ${\displaystyle \frac{S_n}{n}=\frac{\sum _{i=1}^n{X}_i}{n}}$. What value it converges to? What is its distribution? Law of Large Numbers (LLN) and Central Limit Theorem (CLT) will answer these questions.

The requirement of finite variance can be neglected, if we consider a stronger version

1.2 CLT

• The theorem is equivalent to, for large ${\displaystyle n}$, ${\displaystyle \frac{S_n}{n}}$ is well approximated by ${\displaystyle \mathcal{N}(\mu ,\frac{{\sigma }^2}{n})}$.
• We actually perform a standardization $$\frac{\frac{S_n}{n}-\mu }{\frac{\sigma }{\sqrt{n}}}=\frac{\sqrt{n}}{\sigma }(\frac{S_n}{n}-\mu ).$$ This RV has mean ${\displaystyle 0}$ and variance ${\displaystyle 1}$.

1.2.1 Extensions of CLT

If ${\displaystyle X_1,X_2,\cdots }$ are independent but not identically distributed, CLT can apply with additional conditions like Lyapunov or Lindeberg condition. A special case is for bounded RVs. Suppose ${\displaystyle X_1,X_2,\cdots }$ independent, with ${\displaystyle \mathbb{E}[X_i]={\mu }_i<\mathrm{\infty }}$, ${\displaystyle \mathrm{Var}(X_i)={\sigma }_i^2<\mathrm{\infty }}$, ${\displaystyle S_n=X_1+\cdots +X_n}$, then if

1. ${\displaystyle \mathrm{\exists }}$ a constant ${\displaystyle M>0}$: ${\displaystyle \mathbb{P}(|X_i|<M)=1,\mathrm{\forall }i\in \mathbb{N}}$.
2. ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}(S_n)=\underset{n\to \mathrm{\infty }}{lim}\sum _{i=1}^n{\sigma }_i^2=\mathrm{\infty }}$,

we have $$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(\frac{S_n-\mathbb{E}[S_n]}{\sqrt{\mathrm{Var}(S_n)}}<x)=\mathrm{\Phi }(x),\mathrm{\forall }x\in \mathbb{R}.$$

2 Types of Convergence for RV

${\displaystyle X_1,X_2,\cdots }$ is an infinite sequence of RVs. ${\displaystyle X}$ is another RV. Assume they are all defined on the same probability space. Of course we can define pointwise convergence $$\underset{n\to \mathrm{\infty }}{lim}{X}_n(\omega )=X(\omega ),\mathrm{\forall }\omega \in \mathrm{\Omega }.$$
This notion of convergence however, turns out to be too strong.

More precisely, $$\mathbb{P}\left(\{\omega \in \mathrm{\Omega }|\underset{n\to \mathrm{\infty }}{lim}{X}_n(\omega )=X(\omega )\}\right)=1.$$
I.e., ${\displaystyle \mathrm{\forall }\epsilon >0,\mathrm{\exists }N(\omega ,\epsilon )\in \mathbb{N},\mathrm{\forall }n\ge N(\omega ,\epsilon ):|X_n(\omega )-X|<\epsilon }$. This is actually a definition for pointwise convergence.

• Equal in distribution does not mean equal in probability. Take a fair coin and toss it twice. Assume the two tosses are independent. ${\displaystyle \mathrm{\Omega }=\{\text{HH,HT,TH,TT}\}}$. Let ${\displaystyle X_1,X_2}$ be defined as $$X_i(\omega )=\{\begin{array}{rl}& 1,i\text{-th toss shows H},\\ & 0,i\text{-th toss shows T}.\end{array}$$
  Then ${\displaystyle X_1\stackrel{\mathrm{d}}{\to }{X}_2}$, but ${\displaystyle \mathbb{P}(X_1=X_2)=\mathbb{P}(\{\text{HH,TT}\})=\frac{1}{2}}$. So ${\displaystyle X_1,X_2}$ are not equal in probability.
• Example for convergence in distribution: let ${\displaystyle X_1,X_2,\cdots }$ be RVs with ${\displaystyle \mathbb{P}(X_n=\frac{1}{n})=1}$, and ${\displaystyle X:\mathbb{P}(X=0)=1}$. It's easy to see $$\begin{array}{rl}\underset{n\to \mathrm{\infty }}{lim}{F}_{X_n}(x)& =F_X(x),\mathrm{\forall }x\ne 0,\\ \underset{n\to \mathrm{\infty }}{lim}{F}_{X_n}(x)& \ne F_X(x),x=0.\end{array}$$
  But this does not affect convergence in distribution, since ${\displaystyle F_X(x)}$ is not continuous at ${\displaystyle x=0}$. (in definition we want ${\displaystyle C(F_X)}$)

The three convergence theorems correspond to different convergence.

3 Convergence Relations

By Lemma 1, $$0\le \mathbb{E}[|X_n-X{|}^r]\le \mathbb{E}[|X_n-X{|}^s{]}^{\frac{r}{s}},$$ so if ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathbb{E}[|X_n-X{|}^s]=0}$ for RHS, then ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathbb{E}[|X_n-X{|}^r]=0}$.

Suppose ${\displaystyle X_n\stackrel{\mathrm{a}.\mathrm{s}.}{\to }X}$. Since ${\displaystyle \{|X_m-X|<\epsilon ,\mathrm{\forall }m\ge n\}=\bigcap _{m=n}^{\mathrm{\infty }}\{|X_m-X|<\epsilon \}}$ and ${\displaystyle \mathbb{P}(A)\ge \mathbb{P}(B)}$ if ${\displaystyle B\subset A}$, and by Lemma 2 $$1\ge \mathbb{P}(|X_n-X|<\epsilon )\ge \mathbb{P}(|X_m-X|<\epsilon ,\mathrm{\forall }m\ge n)\to 1,$$ so ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|X_n-X|<\epsilon )=1}$.

Since ${\displaystyle X_n\stackrel{\mathrm{p}}{\to }X}$, ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|X-X_n|>\epsilon )=0,\mathrm{\forall }\epsilon >0}$. So case 1 and 2 together imply $$F_X(x-\epsilon )\le \underset{n\to \mathrm{\infty }}{lim inf}{F}_{X_n}(x)\le \underset{n\to \mathrm{\infty }}{lim sup}{F}_{X_n}(x)\le F_X(x+\epsilon ).$$
If ${\displaystyle x\in C(F_X)}$, then $$\underset{\epsilon \to 0}{lim}{F}_X(x-\epsilon )=F_X(s)=\underset{\epsilon \to 0}{lim}{F}_X(x+\epsilon ),$$ then ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}{F}_{X_n}(x)=F_X(x)}$ if ${\displaystyle x\in C(F_X)}$.

4 Rate of Convergence

• This implies uniform convergence.
• ${\displaystyle C}$ does not depend on the distribution of ${\displaystyle X_i}$. As a matter of fact ${\displaystyle 0.4097\le C\le 0.4748}$.